Partial Likelihood Methods for Probability Density Estimation

Partial likelihood (PL) establishes a sufficiently general framework to develop and study statistical properties of nonlinear techniques in signal processing. In [I], we present the theorem by which the fundamental information-theoretic relationship for learning on the P L cost, the equivalence of likelihood maximization and relative entropy minimization, is established. In this paper, we reformulate the theorem to incorporate both the continuous and discrete probability modeling. We further show that, in both cases, the two conditions of the theorem are satisfied for the basic class of probability models, the exponential family, which includes many important network structures that can be effectively used as probability models. Hence we provide the prospect of using the P L cost in a wide class of applications with different models. We also propose several algorithms for learning/estimating the optimal model parameters by P L maximization. We give examples to illustrate the application of our general formulation and the learning algorithms and demonstrate the advantages of learning on the PL cost by simulation results. PARTIAL LIKELIHOOD FORMULATION The partial likelihood (PL) theory [5] is a recent extension of maximum likelihood (ML), and generalizes the ideas of conditional and marginal likelihood. Since it allows for inclusion of dependent observations, missing data and sequential processing, PL provides us with a general probabilistic framework which is very suitable for application to problems in which time-ordering is essential (e.g. time-series problems), or can be conveniently defined [l]. Partial likelihood can be defined as follows: Consider a time series {zn}, n = 1 , 2 , . . . , and its time-dependent covariates {yn}, and define Fn-l = u{l, [+,-1,. . . , 211, [yn,. . , yl]} as a collection of all relevant events upto *RESEARCH SUPPORTED IN PART BY THE NATIONAL SCIENCE FOUNDATION CAREER AWARD, NSF NCR-9703161. 0-7803-5673-X/99/$10.0